Where we left off from the last Systematic Webinar

- Systematic Technical Analysis is easy with TA-Lib / Pandas TA

- Strategy backtesting is also relatively straightforward in Python

- Lets contrast say the output of 2 strategies

- SMA crossover

- 3Inside cdl strategy, only take signals in direction of SMA

SMA XOver

image.png

3Inside Cdl with SMA trend truncation

image-2.png

These types of result lead us to ask can we identify market regimes to help inform our strategy. We will look at a number of approaches to regime detection.

What is market regime?

'Clusters of persistent market conditions'

How many types of market regime are there?

calm, choppy, trending, crashing ....

How long do they last?

key question -

How do we know when they have changed?

key question - too early signal is noisy, too late signal is costly

How do we test the impact of regime on our strategy?

Approach 1 - What TA indicators can we bring to bear?

In [1]:
import refinitiv.data.eikon as ek
import talib
import pandas_ta as pta
import backtrader as bt
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.pyplot import figure
import matplotlib as mpl
import mplfinance as mpf
import datetime as dt
import numpy as np
from sklearn.svm import SVC
from statsmodels.tsa.stattools import adfuller
import warnings
warnings.simplefilter("ignore")
import configparser
cfg = configparser.ConfigParser()
cfg.read('rdp.cfg',encoding='utf-8')
%matplotlib inline
plt.style.use("dark_background")
mpl.style.use("dark_background")
ek.set_app_key(cfg['eikon']['app_key'])
#ek.set_app_key('cb7b5975d52943ff95e8d128fc87f5fcc88476f1')
In [126]:
df1= ek.get_timeseries('ESc1',fields=['OPEN','HIGH','LOW','CLOSE','VOLUME'],start_date='1997-01-01',
                       end_date='06-30-2022',interval='daily') #',

df1.columns=['open','high','low','close','volume'] #
df1
Out[126]:
open high low close volume
Date
2010-08-05 1124.75 1127.75 1115.5 1123.5 1340145
2010-08-06 1123.5 1127.0 1103.75 1119.5 2177376
2010-08-09 1120.0 1126.75 1116.5 1125.5 998906
2010-08-10 1125.25 1125.75 1108.25 1119.75 2029717
2010-08-11 1119.25 1119.75 1084.5 1085.0 2638258
... ... ... ... ... ...
2022-06-24 3785.25 3919.75 3781.25 3916.25 1563128
2022-06-27 3915.0 3948.0 3892.5 3903.75 1497368
2022-06-28 3911.25 3950.0 3821.75 3825.5 1791690
2022-06-29 3828.25 3840.0 3801.25 3821.25 1717770
2022-06-30 3822.75 3825.0 3741.25 3789.5 2394361

3000 rows × 5 columns

In [127]:
df1.dropna(how="any", inplace=True)
len(df1)
Out[127]:
2997

Pandas TA package is very useful tool to create various technical analysis features with ease

In [128]:
df1.ta.indicators()
Pandas TA - Technical Analysis Indicators - v0.3.14b0
Total Indicators & Utilities: 205
Abbreviations:
    aberration, above, above_value, accbands, ad, adosc, adx, alma, amat, ao, aobv, apo, aroon, atr, bbands, below, below_value, bias, bop, brar, cci, cdl_pattern, cdl_z, cfo, cg, chop, cksp, cmf, cmo, coppock, cross, cross_value, cti, decay, decreasing, dema, dm, donchian, dpo, ebsw, efi, ema, entropy, eom, er, eri, fisher, fwma, ha, hilo, hl2, hlc3, hma, hwc, hwma, ichimoku, increasing, inertia, jma, kama, kc, kdj, kst, kurtosis, kvo, linreg, log_return, long_run, macd, mad, massi, mcgd, median, mfi, midpoint, midprice, mom, natr, nvi, obv, ohlc4, pdist, percent_return, pgo, ppo, psar, psl, pvi, pvo, pvol, pvr, pvt, pwma, qqe, qstick, quantile, rma, roc, rsi, rsx, rvgi, rvi, short_run, sinwma, skew, slope, sma, smi, squeeze, squeeze_pro, ssf, stc, stdev, stoch, stochrsi, supertrend, swma, t3, td_seq, tema, thermo, tos_stdevall, trima, trix, true_range, tsi, tsignals, ttm_trend, ui, uo, variance, vhf, vidya, vortex, vp, vwap, vwma, wcp, willr, wma, xsignals, zlma, zscore

Candle Patterns:
    2crows, 3blackcrows, 3inside, 3linestrike, 3outside, 3starsinsouth, 3whitesoldiers, abandonedbaby, advanceblock, belthold, breakaway, closingmarubozu, concealbabyswall, counterattack, darkcloudcover, doji, dojistar, dragonflydoji, engulfing, eveningdojistar, eveningstar, gapsidesidewhite, gravestonedoji, hammer, hangingman, harami, haramicross, highwave, hikkake, hikkakemod, homingpigeon, identical3crows, inneck, inside, invertedhammer, kicking, kickingbylength, ladderbottom, longleggeddoji, longline, marubozu, matchinglow, mathold, morningdojistar, morningstar, onneck, piercing, rickshawman, risefall3methods, separatinglines, shootingstar, shortline, spinningtop, stalledpattern, sticksandwich, takuri, tasukigap, thrusting, tristar, unique3river, upsidegap2crows, xsidegap3methods

ADX Indicator

In [129]:
adx = df1.copy()
adx.ta.adx(high=adx['high'], low=adx['low'], close=adx['close'], length=50, lensig=50, scalar=100, mamode="rma", drift=0, offset=0, append=True)
adx
Out[129]:
open high low close volume ADX_50 DMP_50 DMN_50
Date
2010-08-05 1124.75 1127.75 1115.5 1123.5 1340145 NaN NaN NaN
2010-08-06 1123.5 1127.0 1103.75 1119.5 2177376 NaN NaN NaN
2010-08-09 1120.0 1126.75 1116.5 1125.5 998906 NaN NaN NaN
2010-08-10 1125.25 1125.75 1108.25 1119.75 2029717 NaN NaN NaN
2010-08-11 1119.25 1119.75 1084.5 1085.0 2638258 NaN NaN NaN
... ... ... ... ... ... ... ... ...
2022-06-24 3785.25 3919.75 3781.25 3916.25 1563128 19.451708 18.098263 25.989542
2022-06-27 3915.0 3948.0 3892.5 3903.75 1497368 19.390447 18.462861 25.700672
2022-06-28 3911.25 3950.0 3821.75 3825.5 1791690 19.383420 17.991334 26.453185
2022-06-29 3828.25 3840.0 3801.25 3821.25 1717770 19.391570 17.850777 26.659825
2022-06-30 3822.75 3825.0 3741.25 3789.5 2394361 19.442843 17.548424 27.421722

2997 rows × 8 columns

In [130]:
fig = plt.figure()

ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)
adx[['close']][1500:1700].plot(figsize=(18,15),ax=ax1)
adx[['ADX_50']][1500.plot(figsize=(18,15),ax=ax2) #'DMP_50','DMN_50'
Out[130]:
<AxesSubplot:xlabel='Date'>

White's VHF Indicator

In [140]:
df2 = df1.copy()
df2.ta.vhf(close=df2['close'].astype(float), length=40, drift=0, offset=0, append=True)
df2
Out[140]:
open high low close volume VHF_40
Date
2010-08-05 1124.75 1127.75 1115.5 1123.5 1340145 NaN
2010-08-06 1123.5 1127.0 1103.75 1119.5 2177376 NaN
2010-08-09 1120.0 1126.75 1116.5 1125.5 998906 NaN
2010-08-10 1125.25 1125.75 1108.25 1119.75 2029717 NaN
2010-08-11 1119.25 1119.75 1084.5 1085.0 2638258 NaN
... ... ... ... ... ... ...
2022-06-24 3785.25 3919.75 3781.25 3916.25 1563128 0.240661
2022-06-27 3915.0 3948.0 3892.5 3903.75 1497368 0.249283
2022-06-28 3911.25 3950.0 3821.75 3825.5 1791690 0.257176
2022-06-29 3828.25 3840.0 3801.25 3821.25 1717770 0.259208
2022-06-30 3822.75 3825.0 3741.25 3789.5 2394361 0.257779

2997 rows × 6 columns

In [150]:
fig = plt.figure()

ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)
df2[['close']][1500:1700].plot(figsize=(18,15),ax=ax1)
df2[['VHF_40']][1500:1700].plot(figsize=(18,15),ax=ax2) #'DMP_50','DMN_50'
Out[150]:
<AxesSubplot:xlabel='Date'>
In [147]:
df3 = df1.copy()
#close=None, length=None, bars=None, offset=None,
df3.ta.ebsw(close=df3['close'].astype(float), length=40, bars=10, offset=0, append=True)
df3
Out[147]:
open high low close volume EBSW_40_10
Date
2010-08-05 1124.75 1127.75 1115.5 1123.5 1340145 NaN
2010-08-06 1123.5 1127.0 1103.75 1119.5 2177376 NaN
2010-08-09 1120.0 1126.75 1116.5 1125.5 998906 NaN
2010-08-10 1125.25 1125.75 1108.25 1119.75 2029717 NaN
2010-08-11 1119.25 1119.75 1084.5 1085.0 2638258 NaN
... ... ... ... ... ... ...
2022-06-24 3785.25 3919.75 3781.25 3916.25 1563128 0.749603
2022-06-27 3915.0 3948.0 3892.5 3903.75 1497368 0.921156
2022-06-28 3911.25 3950.0 3821.75 3825.5 1791690 0.961322
2022-06-29 3828.25 3840.0 3801.25 3821.25 1717770 0.816824
2022-06-30 3822.75 3825.0 3741.25 3789.5 2394361 0.452958

2997 rows × 6 columns

In [149]:
fig = plt.figure()

ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)
df3[['close']][1500:1700].plot(figsize=(18,15),ax=ax1)
df3[['EBSW_40_10']][1500:1700].plot(figsize=(18,15),ax=ax2) #'DMP_50','DMN_50'
Out[149]:
<AxesSubplot:xlabel='Date'>
In [ ]:
 
In [ ]:
 
In [144]:
df4 = df1.copy()
df4.ta.chop(close=df4['close'].astype(np.int64), length=50, atr_length =50, scalar=0, drift=0, offset=0, append=True)
df4
Out[144]:
open high low close volume CHOP_50_50_100
Date
2010-08-05 1124.75 1127.75 1115.5 1123.5 1340145 NaN
2010-08-06 1123.5 1127.0 1103.75 1119.5 2177376 NaN
2010-08-09 1120.0 1126.75 1116.5 1125.5 998906 NaN
2010-08-10 1125.25 1125.75 1108.25 1119.75 2029717 NaN
2010-08-11 1119.25 1119.75 1084.5 1085.0 2638258 NaN
... ... ... ... ... ... ...
2022-06-24 3785.25 3919.75 3781.25 3916.25 1563128 43.012474
2022-06-27 3915.0 3948.0 3892.5 3903.75 1497368 43.101047
2022-06-28 3911.25 3950.0 3821.75 3825.5 1791690 43.193788
2022-06-29 3828.25 3840.0 3801.25 3821.25 1717770 43.283053
2022-06-30 3822.75 3825.0 3741.25 3789.5 2394361 43.368992

2997 rows × 6 columns

In [146]:
fig = plt.figure()

ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)
df4[['close']].plot(figsize=(18,15),ax=ax1)
df4[['CHOP_50_50_100']].plot(figsize=(18,15),ax=ax2) 
Out[146]:
<AxesSubplot:xlabel='Date'>

Approach 2 - Changepoint Detection - offline & online approach

Offline

In [3]:
df5 = df1.copy()
In [13]:
import ruptures as rpt
points = np.array(df5['close'])
In [14]:
#Changepoint detection with the Pelt search method
#model="rbf"
algo = rpt.Pelt(model="rbf").fit(points)
result = algo.predict(pen=10)
rpt.display(points, result, figsize=(8, 4))
plt.title('Change Point Detection: Pelt Search Method')
plt.show()  
In [ ]:
#Changepoint detection with the Binary Segmentation search method
#model = "l2"  
algo = rpt.Binseg(model="l2").fit(points)
my_bkps = algo.predict(n_bkps=10)
# show results
rpt.show.display(points, my_bkps, figsize=(8, 4))
plt.title('Change Point Detection: Binary Segmentation Search Method')
plt.show()
In [ ]:
#Changepoint detection with window-based search method
#model = "l2"  
algo = rpt.Window(width=40, model="l2").fit(points)
my_bkps = algo.predict(n_bkps=10)
rpt.show.display(points, my_bkps, figsize=(8, 4))
plt.title('Change Point Detection: Window-Based Search Method')
plt.show()
In [ ]:
#Changepoint detection with dynamic programming search method
#model = "l1"  
algo = rpt.Dynp(model="l1", min_size=3, jump=5).fit(points)
my_bkps = algo.predict(n_bkps=10)
rpt.show.display(points, my_bkps, figsize=(8, 4))
plt.title('Change Point Detection: Dynamic Programming Search Method')
plt.show()

Online CPD using changefinder - Haykaz

In [4]:
points = df4['close']
In [20]:
%run ./regime_detection.ipynb
In [120]:
import changefinder
#import plotly.express as px
cf = changefinder.ChangeFinder()
scores = [cf.update(p) for p in points]

fig = plt.figure()

ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)
df4[['close']].plot(figsize=(18,15),ax=ax1)
pd.DataFrame(scores).plot(figsize=(18,15),ax=ax2) 
Out[120]:
<AxesSubplot:>
In [118]:
mean = np.mean(scores)
std = np.std(scores)
states = []
for score in scores:
    #no vol
    if score > (mean - 2*std) and score < (mean + 2*std):
        states.append(0)
    #low vol
    # elif score > (mean - 2*std) and score < (mean + 2*std):
    #     states.append(1)
    
    #high vol
    else:
        states.append(1)
In [119]:
plot_hidden_states_cf(np.array(states), df4[['close']],2)
Number of observations for State  0 : 2938
Number of observations for State  1 : 108
In [ ]:
 

Approach 3 - Unsupervised Learning Approaches - K-means, Agglomerative, HMM, GMM et al

To capture the dynamic behavior of such time-series, it is often preferable to employ multiple models. Particularly, when attempting to model financial time-series over a set of regimes, the Markov switching model [18], also known as the regime switching model, has been studied and applied. Generally, the Markov switching model is governed by an unobservable state variable and time-varying transition probabilities. Despite the many empirical applications and theoretical developments involving Markov switching models, a necessary condition for the method is the need to know the number of regimes which exist prior to estimating model coefficients. This limits the flexibility of the regime switching model in two contexts. The first being applications where modeling and forecasting the time-series may be undesirable, and identifying the occurrence of the underlying regimes may be all that is sought after. The second consists of contexts where it may be beneficial to directly infer the number of regimes from data.

Data Ingestion and Engineering

In [53]:
trading_instrument = 'ESc1'
instruments = [trading_instrument]
In [54]:
prices = rd.get_history(
                    universe = instruments,
                    fields=['TRDPRC_1'],
                    #fields=['ASIA_CLOSE'],
                    start =  "1997-01-01",
                    end =  "2022-12-05",
).dropna()
prices.sort_values(by = 'Date', inplace = True)
In [55]:
prices.columns.name = trading_instrument
prices = prices.iloc[:, :-1]
In [56]:
prices.iloc[1782] = (prices.iloc[1781] + prices.iloc[1783])/2
prices
Out[56]:
ESc1 ESc1
Date
1997-09-09 934.25
1997-09-10 917.5
1997-09-11 908.75
1997-09-12 924.5
1997-09-15 921.0
... ...
2022-11-29 3962.25
2022-11-30 4093.0
2022-12-01 4077.5
2022-12-02 4066.75
2022-12-05 4006.25

6359 rows × 1 columns

In [57]:
prices_change, prices, split_index = DataEngineering(prices, split_date = '2006-01-01').prepare_data()
In [58]:
prices_change
Out[58]:
ESc1 ESc1 ESc1_close
Date
1997-09-16 0.002605 946.25
1997-09-17 0.005684 943.75
1997-09-18 0.008505 948.25
1997-09-19 0.007633 960.25
1997-09-22 0.009429 965.5
... ... ...
2022-11-29 0.000212 3962.25
2022-11-30 0.004184 4093.0
2022-12-01 0.001903 4077.5
2022-12-02 0.001614 4066.75
2022-12-05 0.001723 4006.25

6354 rows × 2 columns

Regime Detection - Modeling and In Sample testing

In [59]:
regime_detection = RegimeDetection()
In [17]:
clustering = regime_detection.get_regimes_clustering('AgglomerativeClustering')
clustering_states = clustering.fit_predict(prices)
plot_hidden_states(np.array(clustering_states), prices_change[[f'{trading_instrument}_close']],2)
Number of observations for State  0 : 5204
/Users/jasonram/anaconda3/lib/python3.7/site-packages/plotly/graph_objs/_deprecations.py:385: DeprecationWarning:

plotly.graph_objs.Line is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.scatter.Line
  - plotly.graph_objs.layout.shape.Line
  - etc.


Number of observations for State  1 : 1150
In [18]:
clustering = regime_detection.get_regimes_clustering('kmeans')
clustering_states = clustering.fit_predict(prices)
plot_hidden_states(np.array(clustering_states), prices_change[[f'{trading_instrument}_close']],2)
Number of observations for State  0 : 1926
/Users/jasonram/anaconda3/lib/python3.7/site-packages/plotly/graph_objs/_deprecations.py:385: DeprecationWarning:

plotly.graph_objs.Line is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.scatter.Line
  - plotly.graph_objs.layout.shape.Line
  - etc.


Number of observations for State  1 : 4428
In [22]:
gmm_model = regime_detection.get_regimes_gmm(prices)
gmm_states = gmm_model.predict(prices)
plot_hidden_states(np.array(gmm_states), prices_change[[f'{trading_instrument}_close']],2)
Number of observations for State  0 : 5778
Number of observations for State  1 : 576
In [23]:
hmm_model = regime_detection.get_regimes_hmm(prices)
hmm_states = hmm_model.predict(prices)
plot_hidden_states(np.array(hmm_states), prices_change[[f'{trading_instrument}_close']],2)
Number of observations for State  0 : 5177
Number of observations for State  1 : 1177

Regime detection - Out of Sample testing

In [24]:
states_pred_gmm = feed_forward_training('gmm', prices, split_index, 20)
hmm score initial training 3.8676025465374955
0 hmm retrain 3.8679035603156846
20 hmm retrain 3.871291926911526
40 hmm retrain 3.875783423063066
60 hmm retrain 3.8800488957716586
80 hmm retrain 3.8845271961991688
100 hmm retrain 3.885707223186622
120 hmm retrain 3.88796000341653
140 hmm retrain 3.8901219082681457
160 hmm retrain 3.8937891248366587
180 hmm retrain 3.898307545320299
200 hmm retrain 3.903071916912772
220 hmm retrain 3.9078894418529497
240 hmm retrain 3.912431417937228
260 hmm retrain 3.917032972002971
280 hmm retrain 3.921680799262082
300 hmm retrain 3.921584555296615
320 hmm retrain 3.9234619412849376
340 hmm retrain 3.927387509228905
360 hmm retrain 3.9309310715949204
380 hmm retrain 3.9325549802363002
400 hmm retrain 3.9316851823305865
420 hmm retrain 3.932193994299803
440 hmm retrain 3.9341822748701953
460 hmm retrain 3.9356591208213128
480 hmm retrain 3.9326033893759558
500 hmm retrain 3.931248617104375
520 hmm retrain 3.925503069506324
540 hmm retrain 3.9262597660559315
560 hmm retrain 3.9243697197348735
580 hmm retrain 3.9233654753345197
600 hmm retrain 3.9261575536743636
620 hmm retrain 3.927001436892071
640 hmm retrain 3.9255076852857425
660 hmm retrain 3.926668501105242
680 hmm retrain 3.9265008851066776
700 hmm retrain 3.9034127027562064
720 hmm retrain 3.8885976203152293
740 hmm retrain 3.871332405189118
760 hmm retrain 3.866848142316542
780 hmm retrain 3.859960956811511
800 hmm retrain 3.848808765960446
820 hmm retrain 3.8375626130061518
840 hmm retrain 3.836558130720161
860 hmm retrain 3.8339289506702694
880 hmm retrain 3.8341834217804798
900 hmm retrain 3.828135250734329
920 hmm retrain 3.829213228796829
940 hmm retrain 3.828463980827423
960 hmm retrain 3.8297241695381743
980 hmm retrain 3.828776070745655
1000 hmm retrain 3.832255900770075
1020 hmm retrain 3.8356489291127773
1040 hmm retrain 3.833757771082706
1060 hmm retrain 3.836355396677266
1080 hmm retrain 3.8399161070839325
1100 hmm retrain 3.839833851131179
1120 hmm retrain 3.834878087952279
1140 hmm retrain 3.828103658387172
1160 hmm retrain 3.8291683160051795
1180 hmm retrain 3.8273206316043327
1200 hmm retrain 3.8302009034292825
1220 hmm retrain 3.8333107251148
1240 hmm retrain 3.8343183790226263
1260 hmm retrain 3.8371920832702093
1280 hmm retrain 3.8405704445652926
1300 hmm retrain 3.8430728335692055
1320 hmm retrain 3.8443449276690105
1340 hmm retrain 3.8467252663432383
1360 hmm retrain 3.8491600276997144
1380 hmm retrain 3.850481835810161
1400 hmm retrain 3.8501022120293826
1420 hmm retrain 3.8412303713424216
1440 hmm retrain 3.8391505921629308
1460 hmm retrain 3.834003417621304
1480 hmm retrain 3.8345364842487397
1500 hmm retrain 3.8295089400225053
1520 hmm retrain 3.8308190338381793
1540 hmm retrain 3.8335070659215167
1560 hmm retrain 3.8362889175105757
1580 hmm retrain 3.838020158755257
1600 hmm retrain 3.8395304168257565
1620 hmm retrain 3.8394606199077637
1640 hmm retrain 3.8395463885178573
1660 hmm retrain 3.8407558187740447
1680 hmm retrain 3.8437557462817997
1700 hmm retrain 3.845499069670624
1720 hmm retrain 3.846859786399829
1740 hmm retrain 3.8474556505288255
1760 hmm retrain 3.8497739004696094
1780 hmm retrain 3.8515623260105634
1800 hmm retrain 3.854409098307587
1820 hmm retrain 3.856705979779422
1840 hmm retrain 3.858378649907294
1860 hmm retrain 3.860556251282165
1880 hmm retrain 3.862585490523129
1900 hmm retrain 3.8630156616493974
1920 hmm retrain 3.865447589271947
1940 hmm retrain 3.867042671932966
1960 hmm retrain 3.8683137768843494
1980 hmm retrain 3.8703610284251826
2000 hmm retrain 3.873161706923815
2020 hmm retrain 3.875166842263663
2040 hmm retrain 3.8757877034506536
2060 hmm retrain 3.8773816742847713
2080 hmm retrain 3.8792399070053687
2100 hmm retrain 3.8811307819523106
2120 hmm retrain 3.8836928893958427
2140 hmm retrain 3.886343529566604
2160 hmm retrain 3.888737335836215
2180 hmm retrain 3.890359927219905
2200 hmm retrain 3.892601163692457
2220 hmm retrain 3.890766742484769
2240 hmm retrain 3.8927656768300687
2260 hmm retrain 3.8926374490184212
2280 hmm retrain 3.893162852172066
2300 hmm retrain 3.8944647605779767
2320 hmm retrain 3.896001929585878
2340 hmm retrain 3.897969131742552
2360 hmm retrain 3.9005289676111152
2380 hmm retrain 3.902997503972978
2400 hmm retrain 3.9043843512707324
2420 hmm retrain 3.9060954049909857
2440 hmm retrain 3.903803296452687
2460 hmm retrain 3.902107939186225
2480 hmm retrain 3.9038051039212807
2500 hmm retrain 3.90498579376983
2520 hmm retrain 3.9060627022709564
2540 hmm retrain 3.904239057971329
2560 hmm retrain 3.902718166155975
2580 hmm retrain 3.904510873230246
2600 hmm retrain 3.90662776807723
2620 hmm retrain 3.908278528062177
2640 hmm retrain 3.90949250893969
2660 hmm retrain 3.910077881602492
2680 hmm retrain 3.9127980517754555
2700 hmm retrain 3.91454622744565
2720 hmm retrain 3.9168362062823485
2740 hmm retrain 3.9177663811919303
2760 hmm retrain 3.9195456993377262
2780 hmm retrain 3.9220565940660754
2800 hmm retrain 3.924129570422781
2820 hmm retrain 3.92638157640826
2840 hmm retrain 3.9285130407432147
2860 hmm retrain 3.9308427510881323
2880 hmm retrain 3.933014673705131
2900 hmm retrain 3.935552993848433
2920 hmm retrain 3.9380254323545354
2940 hmm retrain 3.9402053169463924
2960 hmm retrain 3.9426320120014298
2980 hmm retrain 3.945189569777396
3000 hmm retrain 3.947622267739755
3020 hmm retrain 3.9499400178568775
3040 hmm retrain 3.951398733315915
3060 hmm retrain 3.94901642323452
3080 hmm retrain 3.9484988523938145
3100 hmm retrain 3.949295006599404
3120 hmm retrain 3.950903337499761
3140 hmm retrain 3.952815240622698
3160 hmm retrain 3.954119299133171
3180 hmm retrain 3.9562945303108945
3200 hmm retrain 3.958356602719114
3220 hmm retrain 3.958446876520191
3240 hmm retrain 3.957218992732351
3260 hmm retrain 3.9545273632784244
3280 hmm retrain 3.951218304430024
3300 hmm retrain 3.9522485558989526
3320 hmm retrain 3.9539640938879366
3340 hmm retrain 3.955311195066335
3360 hmm retrain 3.9574742133711296
3380 hmm retrain 3.9576659113303116
3400 hmm retrain 3.9581629787402792
3420 hmm retrain 3.9599989609231288
3440 hmm retrain 3.959331256851102
3460 hmm retrain 3.9606184905059534
3480 hmm retrain 3.961611675225664
3500 hmm retrain 3.963737511115142
3520 hmm retrain 3.965525636653504
3540 hmm retrain 3.967616664141009
3560 hmm retrain 3.968613168212244
3580 hmm retrain 3.956186037337353
3600 hmm retrain 3.948874345444055
3620 hmm retrain 3.9477859558261
3640 hmm retrain 3.944637470716382
3660 hmm retrain 3.9439720382304273
3680 hmm retrain 3.9451485126119055
3700 hmm retrain 3.9447002728938245
3720 hmm retrain 3.944034433129027
3740 hmm retrain 3.9416226779579735
3760 hmm retrain 3.9410167488753602
3780 hmm retrain 3.942933423676253
3800 hmm retrain 3.9434646436594636
3820 hmm retrain 3.9437985200069625
3840 hmm retrain 3.944252036772842
3860 hmm retrain 3.94558846178192
3880 hmm retrain 3.947090532666623
3900 hmm retrain 3.949103931578617
3920 hmm retrain 3.950480518255425
3940 hmm retrain 3.9524911105768834
3960 hmm retrain 3.9536585801682
3980 hmm retrain 3.9541086971763524
4000 hmm retrain 3.955445138788695
4020 hmm retrain 3.955589787472933
4040 hmm retrain 3.9559536169651937
4060 hmm retrain 3.9535587671488535
4080 hmm retrain 3.952205294506467
4100 hmm retrain 3.950784340011034
4120 hmm retrain 3.948977610125703
4140 hmm retrain 3.94709716688135
4160 hmm retrain 3.9434472708862263
4180 hmm retrain 3.9425686610964226
4200 hmm retrain 3.9405967450920882
4220 hmm retrain 3.9381646303563977
4240 hmm retrain 3.936558951189287
4260 hmm retrain 3.9355366481922855
In [25]:
plot_hidden_states(np.array(states_pred_gmm), prices_change[[f'{trading_instrument}_close']][split_index:],2)
Number of observations for State  0 : 2747
Number of observations for State  1 : 1518
In [60]:
states_pred_hmm = feed_forward_training('hmm', prices, split_index, 20)
hmm score initial training 8177.390611705038
0 hmm retrain 8181.963393327508
20 hmm retrain 8268.323382548859
40 hmm retrain 8357.125733396171
60 hmm retrain 8445.768437413812
80 hmm retrain 8534.928372031323
100 hmm retrain 8616.029707426418
120 hmm retrain 8700.22015340714
140 hmm retrain 8784.517878342887
160 hmm retrain 8872.135803112784
180 hmm retrain 8961.877639168266
200 hmm retrain 9052.730331031456
220 hmm retrain 9143.29627843573
240 hmm retrain 9233.526315880716
260 hmm retrain 9324.20929495686
280 hmm retrain 9415.126753696657
300 hmm retrain 9492.261969519279
320 hmm retrain 9576.631654340441
340 hmm retrain 9666.70645217337
360 hmm retrain 9755.597653330198
380 hmm retrain 9839.780665920185
400 hmm retrain 9916.398579049745
420 hmm retrain 9995.457255738538
440 hmm retrain 10080.736734034399
460 hmm retrain 10164.060004534174
480 hmm retrain 10233.421027270537
500 hmm retrain 10308.846619015381
520 hmm retrain 10376.104677292527
540 hmm retrain 10456.416612625268
560 hmm retrain 10528.409250922618
580 hmm retrain 10605.672139061353
600 hmm retrain 10693.3880197724
620 hmm retrain 10773.811417596868
640 hmm retrain 10846.711363450046
660 hmm retrain 10929.328207465564
680 hmm retrain 11006.114705898266
700 hmm retrain 11047.861286393072
720 hmm retrain 11103.103319395917
740 hmm retrain 11152.134805465394
760 hmm retrain 11217.642704470374
780 hmm retrain 11284.125395071425
800 hmm retrain 11345.735269675992
820 hmm retrain 11400.413338987215
840 hmm retrain 11473.817072221676
860 hmm retrain 11542.682746317507
880 hmm retrain 11621.238227724822
900 hmm retrain 11683.595440270805
920 hmm retrain 11764.858478155567
940 hmm retrain 11840.21317612793
960 hmm retrain 11922.48442353193
980 hmm retrain 11995.614770890106
1000 hmm retrain 12085.657802329533
1020 hmm retrain 12174.497777319546
1040 hmm retrain 12245.056888880727
1060 hmm retrain 12331.598078243376
1080 hmm retrain 12421.473252894373
1100 hmm retrain 12498.214688566208
1120 hmm retrain 12564.06283727226
1140 hmm retrain 12628.132064595722
1160 hmm retrain 12708.07589459429
1180 hmm retrain 12777.99833949063
1200 hmm retrain 12866.24434135288
1220 hmm retrain 12955.406753670679
1240 hmm retrain 13037.0991268553
1260 hmm retrain 13125.674225704459
1280 hmm retrain 13216.183147681599
1300 hmm retrain 13303.746843675948
1320 hmm retrain 13386.22081725211
1340 hmm retrain 13473.495233048447
1360 hmm retrain 13561.080091507201
1380 hmm retrain 13643.461126576252
1400 hmm retrain 13719.70105368182
1420 hmm retrain 13779.727216856598
1440 hmm retrain 13849.580515849522
1460 hmm retrain 13911.929536956963
1480 hmm retrain 13990.005389860476
1500 hmm retrain 14054.435822735939
1520 hmm retrain 14136.908878386903
1540 hmm retrain 14225.877781076071
1560 hmm retrain 14315.13287109597
1580 hmm retrain 14399.785393130865
1600 hmm retrain 14483.737062067621
1620 hmm retrain 14559.410521763535
1640 hmm retrain 14637.920539380599
1660 hmm retrain 14721.114576494947
1680 hmm retrain 14811.599797337694
1700 hmm retrain 14897.334851009513
1720 hmm retrain 14980.808100617618
1740 hmm retrain 15060.9601068476
1760 hmm retrain 15148.560722429853
1780 hmm retrain 15234.303091174326
1800 hmm retrain 15324.676702831173
1820 hmm retrain 15413.181763229823
1840 hmm retrain 15498.757174939894
1860 hmm retrain 15587.023968834454
1880 hmm retrain 15674.239119436003
1900 hmm retrain 15753.381448603746
1920 hmm retrain 15842.421460816411
1940 hmm retrain 15928.152675062076
1960 hmm retrain 16012.425755690783
1980 hmm retrain 16100.377438578791
2000 hmm retrain 16191.750128248948
2020 hmm retrain 16279.830543032587
2040 hmm retrain 16360.35658015524
2060 hmm retrain 16446.529962967234
2080 hmm retrain 16534.01270698536
2100 hmm retrain 16621.40623964643
2120 hmm retrain 16712.2895830002
2140 hmm retrain 16803.607080326732
2160 hmm retrain 16893.50282369965
2180 hmm retrain 16980.194639850994
2200 hmm retrain 17069.729818243195
2220 hmm retrain 17140.424908830144
2240 hmm retrain 17228.17222779781
2260 hmm retrain 17305.452355309735
2280 hmm retrain 17384.87653838438
2300 hmm retrain 17470.50372834322
2320 hmm retrain 17556.9003339278
2340 hmm retrain 17645.637290117407
2360 hmm retrain 17737.44078203952
2380 hmm retrain 17828.80525148944
2400 hmm retrain 17914.69417075017
2420 hmm retrain 18002.368172182516
2440 hmm retrain 18073.670902755013
2460 hmm retrain 18144.395526220815
2480 hmm retrain 18230.928495392844
2500 hmm retrain 18315.599283626267
2520 hmm retrain 18399.68455726063
2540 hmm retrain 18474.791949856244
2560 hmm retrain 18544.90514646624
2580 hmm retrain 18633.911047813956
2600 hmm retrain 18724.439916793344
2620 hmm retrain 18812.644768584352
2640 hmm retrain 18897.19818813259
2660 hmm retrain 18979.190812958575
2680 hmm retrain 19073.04376245287
2700 hmm retrain 19161.26546946402
2720 hmm retrain 19252.94709097933
2740 hmm retrain 19337.353431074745
2760 hmm retrain 19427.21024392244
2780 hmm retrain 19520.500491934443
2800 hmm retrain 19613.33331650544
2820 hmm retrain 19705.959772985036
2840 hmm retrain 19797.258190834615
2860 hmm retrain 19890.23247052671
2880 hmm retrain 19983.014033609245
2900 hmm retrain 20076.911625181136
2920 hmm retrain 20171.729341488168
2940 hmm retrain 20264.215345670415
2960 hmm retrain 20359.419985086664
2980 hmm retrain 20455.59508725137
3000 hmm retrain 20551.26058513357
3020 hmm retrain 20647.1295470958
3040 hmm retrain 20739.145648756923
3060 hmm retrain 20813.585079062486
3080 hmm retrain 20893.76337159298
3100 hmm retrain 20979.0622229486
3120 hmm retrain 21070.258595180567
3140 hmm retrain 21163.554991041045
3160 hmm retrain 21252.10511366261
3180 hmm retrain 21347.263803759
3200 hmm retrain 21442.256913017267
3220 hmm retrain 21530.077294359675
3240 hmm retrain 21607.109480011626
3260 hmm retrain 21677.572134642753
3280 hmm retrain 21746.049940570712
3300 hmm retrain 21834.006917886774
3320 hmm retrain 21926.292923373498
3340 hmm retrain 22017.296999962105
3360 hmm retrain 22112.74504903318
3380 hmm retrain 22188.271822151106
3400 hmm retrain 22272.994902298553
3420 hmm retrain 22365.97503254564
3440 hmm retrain 22442.56321128738
3460 hmm retrain 22532.813816884063
3480 hmm retrain 22618.447641338684
3500 hmm retrain 22715.55810847475
3520 hmm retrain 22809.748328141563
3540 hmm retrain 22906.7142482936
3560 hmm retrain 22993.418890863668
3580 hmm retrain 23013.809062034925
3600 hmm retrain 23052.702048925137
3620 hmm retrain 23122.12614648981
3640 hmm retrain 23186.363493956862
3660 hmm retrain 23261.978197704724
3680 hmm retrain 23352.683342099557
3700 hmm retrain 23435.144397873366
3720 hmm retrain 23513.083390715325
3740 hmm retrain 23580.129110342925
3760 hmm retrain 23660.654701147538
3780 hmm retrain 23755.167342592944
3800 hmm retrain 23837.720679704475
3820 hmm retrain 23917.532283987854
3840 hmm retrain 24000.43846626147
3860 hmm retrain 24092.260820795
3880 hmm retrain 24182.5982179506
3900 hmm retrain 24278.090010386743
3920 hmm retrain 24369.371492787282
3940 hmm retrain 24464.98404386143
3960 hmm retrain 24552.6187515478
3980 hmm retrain 24633.54593953177
4000 hmm retrain 24725.211049179474
4020 hmm retrain 24803.592437468258
4040 hmm retrain 24883.454519815346
4060 hmm retrain 24954.936094826804
4080 hmm retrain 25027.476804882703
4100 hmm retrain 25097.079024958333
4120 hmm retrain 25169.94884135688
4140 hmm retrain 25239.936513573608
4160 hmm retrain 25302.546014351366
4180 hmm retrain 25374.08375730386
4200 hmm retrain 25449.29368814844
4220 hmm retrain 25521.69833728425
4240 hmm retrain 25590.488785775433
4260 hmm retrain 25664.575355634668
In [27]:
plot_hidden_states(np.array(states_pred_hmm), prices_change[[f'{trading_instrument}_close']][split_index:],2)
Number of observations for State  0 : 3795
Number of observations for State  1 : 470
In [ ]:
 

Strategy Implementation

In [61]:
prices_open = rd.get_history(
                    universe = 'ESc1',
                    fields=[ 'OPEN_PRC'],
                    start =  "1997-09-09",
                    end =  "2022-11-24",).dropna()

prices_open.sort_values(by = 'Date', inplace = True)
prices_open = prices_open.iloc[:, :-1]
prices_open.columns.name = trading_instrument
In [62]:
prices_open = prices_open[split_index:]
prices_open = prices_open.shift(-1)
prices_open.head()
Out[62]:
ESc1 ESc1
Date
2005-12-23 1277.0
2005-12-27 1263.5
2005-12-28 1265.25
2005-12-29 1260.25
2005-12-30 1255.25
In [63]:
prices = pd.DataFrame(prices_change[split_index:][f'{trading_instrument}_close'])
prices['State'] = states_pred_hmm
In [64]:
prices = pd.DataFrame(prices['State']).merge(prices_open, on = 'Date')
prices.reset_index(inplace=True)
prices.dropna(inplace=True)
prices.columns.name = trading_instrument
prices
Out[64]:
ESc1 Date State ESc1
0 2006-01-03 0 1274.5
1 2006-01-04 0 1280.5
2 2006-01-05 0 1281.25
3 2006-01-06 0 1291.25
4 2006-01-09 0 1294.25
... ... ... ...
4252 2022-11-16 1 3977.25
4253 2022-11-17 1 3960.5
4254 2022-11-18 1 3972.5
4255 2022-11-21 0 3957.75
4256 2022-11-22 0 4010.0

4257 rows × 3 columns

HMM Strategy

In [65]:
strategy_outcome = HMMStrategy(prices, 2).run()
strategy_outcome
Out[65]:
Current Price Signal State Position Stock Balance Cash Balance Stock Value Portfolio Valuation P&L
Date
2006-01-03 1274.50 1 0 Idle 0 0.0 0.00 0.00 0.0
2006-01-04 1280.50 1 0 Buy 1 -1280.5 1280.50 0.00 0.0
2006-01-05 1281.25 1 0 Hold 1 -1280.5 1281.25 0.75 0.0
2006-01-06 1291.25 1 0 Hold 1 -1280.5 1291.25 10.75 0.0
2006-01-09 1294.25 1 0 Hold 1 -1280.5 1294.25 13.75 0.0
... ... ... ... ... ... ... ... ... ...
2022-11-16 3977.25 0 1 Idle 0 2324.5 0.00 2324.50 2324.5
2022-11-17 3960.50 0 1 Idle 0 2324.5 0.00 2324.50 2324.5
2022-11-18 3972.50 0 1 Idle 0 2324.5 0.00 2324.50 2324.5
2022-11-21 3957.75 1 0 Idle 0 2324.5 0.00 2324.50 2324.5
2022-11-22 4010.00 1 0 Buy 1 -1685.5 4010.00 2324.50 2324.5

4257 rows × 9 columns

In [66]:
strategy_outcome['Current Price'] = strategy_outcome['Current Price'] - 1278.25
StrategyOutcomePlot(strategy_outcome).plot(outcome = 'P&L', signal_dot = 'Current Price')

MA Strategy with HMM

In [34]:
strategy_outcome = MovingaAveragesStrategy(prices, 10, 30, True, 2).run()
strategy_outcome
Out[34]:
Current Price ESc1_fast_ma ESc1_slow_ma Crossover Signal State Position Stock Balance Cash Balance Stock Value Portfolio Valuation P&L
Date
2006-02-14 1278.25 1269.650 1278.908333 0 NaN 0 Idle 0 0.0 0.0 0.0 0.0
2006-02-15 1282.00 1269.175 1279.158333 0 0.0 0 Idle 0 0.0 0.0 0.0 0.0
2006-02-16 1291.75 1271.175 1279.533333 0 0.0 0 Idle 0 0.0 0.0 0.0 0.0
2006-02-17 1288.25 1273.425 1279.766667 0 0.0 0 Idle 0 0.0 0.0 0.0 0.0
2006-02-21 1285.25 1275.100 1279.566667 0 0.0 0 Idle 0 0.0 0.0 0.0 0.0
... ... ... ... ... ... ... ... ... ... ... ... ...
2022-11-16 3977.25 3877.725 3786.525000 1 0.0 1 Idle 0 1421.0 0.0 1421.0 1421.0
2022-11-17 3960.50 3901.375 3793.683333 1 0.0 1 Idle 0 1421.0 0.0 1421.0 1421.0
2022-11-18 3972.50 3923.625 3804.808333 1 0.0 1 Idle 0 1421.0 0.0 1421.0 1421.0
2022-11-21 3957.75 3938.125 3815.908333 1 0.0 0 Idle 0 1421.0 0.0 1421.0 1421.0
2022-11-22 4010.00 3955.875 3829.341667 1 0.0 0 Idle 0 1421.0 0.0 1421.0 1421.0

4228 rows × 12 columns

In [35]:
strategy_outcome['Current Price'] = strategy_outcome['Current Price'] - 1278.25
StrategyOutcomePlot(strategy_outcome).plot(outcome = 'P&L', signal_dot = 'Current Price')
In [67]:
strategy_outcome_hmm = HMMStrategy(prices,3).run()
strategy_outcome_ma = MovingaAveragesStrategy(prices, 10, 30, False, 2).run()
strategy_outcome_ma_hmm = MovingaAveragesStrategy(prices, 10, 30, True, 2).run()
In [68]:
strategies = strategy_outcome_hmm[['Current Price', 'Portfolio Valuation', 'P&L']].merge(
                strategy_outcome_ma[['Current Price','Portfolio Valuation', 'P&L']].merge(
                strategy_outcome_ma_hmm[['Current Price','Portfolio Valuation', 'P&L']], on = 'Date', suffixes = ['_ma', '_ma_hmm']), on = 'Date')
strategies['Buy and Hold'] = strategies['Current Price'] - strategies['Current Price'][0]
strategies
Out[68]:
Current Price Portfolio Valuation P&L Current Price_ma Portfolio Valuation_ma P&L_ma Current Price_ma_hmm Portfolio Valuation_ma_hmm P&L_ma_hmm Buy and Hold
Date
2006-02-14 1278.25 -2.25 0.0 1278.25 0.00 0.0 1278.25 0.0 0.0 0.00
2006-02-15 1282.00 1.50 0.0 1282.00 0.00 0.0 1282.00 0.0 0.0 3.75
2006-02-16 1291.75 11.25 0.0 1291.75 0.00 0.0 1291.75 0.0 0.0 13.50
2006-02-17 1288.25 7.75 0.0 1288.25 0.00 0.0 1288.25 0.0 0.0 10.00
2006-02-21 1285.25 4.75 0.0 1285.25 0.00 0.0 1285.25 0.0 0.0 7.00
... ... ... ... ... ... ... ... ... ... ...
2022-11-16 3977.25 2734.50 2734.5 3977.25 974.50 841.5 3977.25 1421.0 1421.0 2699.00
2022-11-17 3960.50 2734.50 2734.5 3960.50 957.75 841.5 3960.50 1421.0 1421.0 2682.25
2022-11-18 3972.50 2734.50 2734.5 3972.50 969.75 841.5 3972.50 1421.0 1421.0 2694.25
2022-11-21 3957.75 2734.50 2734.5 3957.75 955.00 841.5 3957.75 1421.0 1421.0 2679.50
2022-11-22 4010.00 2734.50 2734.5 4010.00 1007.25 841.5 4010.00 1421.0 1421.0 2731.75

4228 rows × 10 columns

In [70]:
fig = make_subplots(specs=[[{"secondary_y": True}]])

fig.add_trace(go.Line(x=strategies.index, y=strategies["Buy and Hold"], name = 'Buy and Hold',
            line_color = 'black'),secondary_y=True)

fig.add_trace(go.Line(x= strategy_outcome.index, y=strategies['P&L_ma_hmm'], name = 'P&L for MA with hmm',
            line_color = 'khaki'),  secondary_y=False)

fig.add_trace(go.Line(x= strategy_outcome.index, y=strategies['P&L_ma'], name = 'P&L for MA without hmm',
            line_color = 'grey'),  secondary_y=False)

fig.add_trace(go.Line(x= strategy_outcome.index, y=strategies['P&L'], name = 'P&L HMM',
            line_color = 'green'),  secondary_y=False)

fig.add_trace(go.Line(x= strategy_outcome.index, y=strategies['Portfolio Valuation_ma_hmm'], name = 'Valuation for MA with hmm',
            line_color = 'cyan'),  secondary_y=True)

fig.add_trace(go.Line(x= strategy_outcome.index, y=strategies['Portfolio Valuation_ma'], name = 'Valuation for MA without hmm',
            line_color = 'blue'),  secondary_y=True)

fig.add_trace(go.Line(x= strategy_outcome.index, y=strategies['Portfolio Valuation'], name = 'Valuation HMM',
            line_color = 'brown'),  secondary_y=True)
fig.update_layout(height=400, width=900, legend=dict(
    yanchor="top", y=0.99,
    xanchor="left",x=0.01), 
    margin=dict(l=20, r=20, t=20, b=20))

fig.show()
In [ ]: